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Editorial 


Among the many challenges that must be overcome in 
successfully developing a novel drug treatment is to 
understand and prepare evidence that will ultimately be 
required by regulators and reimbursement authorities. 
Payers and other stakeholders often state the need for 
evidence of the comparative effectiveness of new 
treatments, in some cases arguing that this evidence 
should be a standard requirement for market access and 
reimbursement. Comparative effectiveness research 
(CER) is a method of developing evidence of the clinical 
and cost effectiveness of true alternative interventions, 
potentially including long-term data that assess patient- 
centered outcomes and evidence that a new treatment 
actually changes clinical practice. Conceiving of and 
executing a plan for developing CER to support these 
evidence requirements is therefore a_ substantial 
challenge, one made even more difficult by evolving 
notions of CER and uncertainty in what constitutes 
sufficient evidence of comparative effectiveness. In fact, 
CER as a methodology is rapidly evolving, incorporating 
new sources of real-world and big data and applying 
cutting edge analytic techniques including novel 
statistical methods and machine learning [1]. This review 
aims to analyze the evolving conception of CER in order to 
support researchers and others in the life sciences 
industry in planning and designing a clinical development 
program to meet potential future requirements for CER 
that may increasingly determine commercial success. 


The Evolving Definition of Comparative Effectiveness Research: Lessons for 


Researchers Planning and Designing a Clinical Development Program 


Early definitions of CER are often associated with the 
heightened attention and new funding that came with the 
American Recovery and Reinvestment Act (ARRA) of 
2009 and with the establishment of the Patient-Centered 
Outcomes Research Institute (PCORI). Policy documents 
that established this view of CER argued for the 
importance of large prospective studies that could 
definitively establish the best (most effective) course of 
treatment [2-4]. More recently, definitions of CER have 
evolved to become more operational than theoretical, 
describing CER as a tool for a specific purpose; namely, to 
provide the specific evidence needed by clinicians, 
patients, policymakers, health plans, and other payers to 
make specific treatment and resource allocation decisions 
[5,6]. 


One example that arguably fits the original conception 
of CER is the widely-disseminated study of the 
comparative effectiveness of percutaneous coronary 
intervention (PCI) versus drug therapy alone in patients 
with stable angina (COURAGE study) [7,8]. In this study, a 
modest symptom benefit of PCI relative to drug therapy 
was observed, but no evidence was found for improved 
survival or reduction in the rate of myocardial infarction, 
an absence of effect that persisted over 15 years of follow 
up [9]. The observed clinical benefit meant that PCI 
demonstrated comparative clinical effectiveness, while 
the small magnitude of the benefit and its incremental 
cost meant that it was not cost effective at typically 
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accepted thresholds [10]. Moreover, the effect of this new 
evidence was shown to have a real impact on clinical 
practice [11]. The full potential of CER is exemplified by 
studies such as COURAGE (and its successor, ISCHEMIA) 
[12], and by several others, [13-15], however, the number 
of clear-cut success stories for large centralized 
comparative clinical trials has arguably been modest, 
especially considering the vast number of treatment and 
resource allocation decisions where evidence is sorely 
needed. 


An emerging definition of CER is quite distinct from the 
approach taken in these large, broadly inclusive clinical 
trials. Real-world, observational data often provide 
evidence of comparative effectiveness, potentially from a 
range of sources designed to meet the needs of different 
decision makers, and thus can provide incremental 
evidence, often supportive or correlative rather than 
independently definitive. Widely-accepted definitions of 
real-world evidence are broader than the original scope of 
CER, including for example electronic health records and 
medical chart reviews, administrative data, and surveys 
[16]. While prospective randomized controlled trials are 
undoubtedly the gold standard for establishing safety and 
efficacy, observational data sources have the potential to 
meet pressing needs of decision makers: real-world 
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populations are typically broadly generalizable, may 
capture relevant outcomes (both clinical and economic) 
and, perhaps most importantly, are widely accessible, 
such that relevant evidence can be generated on a time 
scale of months instead of years. The need for better 
generalizability of evidence reflects the current state of 
evidence for many interventions, particularly for drug 
treatments. It is widely recognized that participants in 
clinical trials submitted to the Food and Drug 
Administration (FDA) for regulatory approval are 
younger and have fewer co-morbid conditions than 
patients in the general population [17-19]. Although there 
are widely understood limitations to observational 
studies including selection bias and unobserved 
confounders, oftentimes real-world observational studies 
are the most appropriate means to validate and expand 
upon the results of registration clinical trials, in particular 
to assess whether benefits observed in registration 
studies are generalizable to real-world populations. 


Publication analysis provides some support for such an 
evolving conception of CER. PubMed citation analysis 
shows a leveling off in the number publications that 
mention CER following the ARRA (after about 2012), 
while interest in ‘real-world’ studies as measured by 
publication counts continues to accelerate (Figure 1). 
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Figure 1: PubMed citation analysis: Comparative effectiveness research versus real-world 


Source: Analysis Group Research. The NCBI PubMed database was searched using the terms 
“real world” and “comparative effectiveness” in title and abstract fields on 7/25/2017. 
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The number of comparative effectiveness clinical trials 
in PubMed has also peaked (Figure 2), with roughly the 
same number of registered CER studies initiated in 2016 
as earlier in the decade. One interpretation of this data is 


that the demand for evidence provided by CER continues 
to grow while the capacity to execute prospective clinical 
trials that assess comparative effectiveness remains 
relatively limited. 
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Figure 2: Clinical trials analysis of comparative effectiveness research studies. 


Source: Analysis Group Research. Clinicaltrials.gov was searched using the term 
“comparative effectiveness" on 7/25/2017. 


The implication for researchers is that there is a 
growing reliance on real-world observational studies to 
supply evidence of comparative effectiveness to meet the 
needs of decision makers. Powerful, traditional CER study 
designs (large, broadly inclusive prospective 
interventional studies comparing two _ alternative 
interventions) may find use primarily for “big picture” 
questions, typically highly prevalent conditions that 
attract academic interest and substantial government 
funding. While prospective interventional studies 
undoubtedly provide valuable data with superior internal 
validity, the future of CER will reflect the accessibility of 
observational study designs and the growing power of 
large databases used for observational studies (aka, big 
data). 


The 21st Century Cures Act has been approved with 
bipartisan support in the lame-duck session of the 114 
United States Congress. This legislation includes 
substantial changes to regulatory processes for drugs and 
medical devices, including provisions for the use of real- 
world observational data to support initial FDA approval 
and to satisfy post-marketing commitments. Clearly, there 
is a movement towards a more pragmatic application of 
real-world data to meet the needs of both regulators and 
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of payers. CER, as it was conceived a decade ago, was 
framed in a context where the availability of real-world 
data and methods for analysis were relatively limited, and 
thus the place of prospective clinical trials to assess 
comparative effectiveness was central. In the next decade, 
as the availability and interconnectedness of real-world 
data increases, comparisons of clinical and _ cost 
effectiveness using real-world data are likely to become 
the standard, with growing influence on the commercial 
success of new health technologies. 
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